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FOREWORD 


The  rapid  increase  in  operational  tempo  and  the  influx  of  sophisticated 
information  technology  systems  into  the  Army  place  huge  demands  on  the  information 
processing  capabilities  of  soldiers  and  their  leaders.  Establishing  and  enhancing 
situation  awareness  (SA)  is  necessary  to  help  these  warfighters  deal  effectively  with  the 
information  demands  of  the  modern  battlefield. 

As  these  trends  continue  and  the  Army  moves  toward  the  Objective  Force, 
psychometrically  sound  measures  of  SA  are  vitally  needed  to  evaluate  the  impact  new 
systems  have  on  SA  and  decision-making.  Many  existing  SA  measures  are  too 
obtrusive  to  use  in  field  exercises  and  other  training  events  that  test  new  systems.  The 
current  research  focused  on  testing  the  Mission  Awareness  Rating  Scale  (MARS),  a 
subjective  measure  of  SA  designed  to  quickly  and  unobtrusively  estimate  SA  through 
self-ratings. 

The  data  reported  here  were  collected  in  conjunction  with  one  of  a  series  of 
experiments  designed  to  investigate  new  ways  to  apply  virtual  environment  technology 
in  training  dismounted  infantry  soldiers  and  small  unit  leaders.  These  experiments  are 
part  of  an  ongoing  four  year  Science  and  Technology  Objective  (STO)  initiative  entitled 
Virtual  Environments  for  Dismounted  Soldier  Simulation,  Training,  and  Mission 
Rehearsal.  The  STO  is  a  collaborative  effort  involving  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences,  the  U.S.  Army  Research  Laboratory,  and  the 
U.S.  Army  Simulation,  Training,  and  Instrumentation  Command. 

This  research  represents  one  of  several  approaches  to  measuring  SA  that  are 
being  developed  and  validated  by  ARI.  The  goal  is  to  provide  researchers  and  trainers 
a  relatively  simple  and  useable  tool  for  measuring  SA  in  a  variety  of  settings.  The 
results  were  briefed  to  the  Department  of  Behavioral  Sciences  and  Leadership,  U.S. 
Military  Academy,  25  February  2002. 


MICHAEL  G.  RUMSEY 


Acting  Technical  Director 
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SITUATION  AWARENESS  IN  A  VIRTUAL  ENVIRONMENT :  DESCRIPTION  OF  A 
SUBJECTIVE  ASSESSMENT  SCALE 

EXECUTIVE  SUMMARY  _ 


Research  Requirement: 

This  research  is  part  of  an  ongoing  effort  by  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences,  Infantry  Forces  Research  Unit,  Fort  Benning, 
Georgia,  to  develop  psychometrically  sound  measures  of  situation  awareness  (SA).  In 
particular,  there  is  a  need  for  SA  measures  that  are  easy  and  practical  to  administer  yet 
provide  useful  SA  information  to  researchers  and  trainers. 


Procedure: 

Sixteen  Fort  Benning  enlisted  soldiers  were  assigned  to  one  of  four  4-man  fire 
teams.  Each  fire  team  completed  four  night  urban  combat  missions  in  a  virtual  reality 
simulator  located  at  the  Land  Warrior  Test  Bed  at  Fort  Benning.  In  each  of  the  four 
missions,  a  different  method  for  simulating  night  vision  goggle  effects  was  used.  After 
each  mission,  each  soldier  completed  the  Mission  Awareness  Rating  Scale  (MARS),  an 
instrument  designed  to  assess  the  subjective  SA  and  workload  experienced  during  a 
given  mission. 


Findings: 

The  results  indicated  that  MARS  discriminated  among  the  four  methods  for 
simulating  night  vision  goggle  environments.  Moreover,  the  night  simulation  method 
identified  by  MARS  as  being  most  similar  to  real  night  operations  was  also  identified  as 
such  through  interviews  with  the  soldiers  involved  in  the  experiment.  This  also 
corresponded  with  their  ratings  of  the  difficulty  of  maintaining  SA  and  workload 
demands  based  on  their  own  experience  in  actual  night  operations. 


Utilization  of  Findings: 

The  results  suggest  that  MARS  has  promise  for  providing  quick  and  relatively 
unobtrusive  measures  of  SA.  Such  a  measure  is  of  particular  utility  in  field  training 
exercises,  where  more  obtrusive  SA  measures  commonly  used  during  simulations 
would  be  viewed  as  overly  disruptive. 
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SITUATION  AWARENESS  IN  A  VIRTUAL  ENVIRONMENT:  DESCRIPTION 
OF  A  SUBJECTIVE  ASSESSMENT  SCALE 

Introduction 

Situation  awareness  (SA)  is  a  construct  that  links  a  variety  of  cognitive 
processes,  such  as  perception,  memory,  and  schemas,  to  decision-making  and 
ultimately  task  performance.  High  levels  of  SA  are  viewed  as  necessary,  but  not 
sufficient,  for  effective  decision-making  and  performance.  Historically,  SA  has  been  of 
most  interest  in  technologically  advanced  domains,  such  as  aviation,  where  a  complex 
physical  environment,  highly  sophisticated  information  and  command  and  control 
systems,  and  task  demands  interact  to  challenge  the  human  operator’s  ability  to 
manage  cognitive  workload  and  make  quick,  effective  decisions.  For  similar  reasons, 
the  Army  is  increasingly  interested  in  SA  as  it  integrates  digital  technologies  into  its 
ground  forces. 

Endsley  (1988)  defined  situation  awareness  (SA)  as  “...the  perception  of  the 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of 
their  meaning  and  the  projection  of  their  status  in  the  near  future.”  Implicit  in  this 
definition  is  the  notion  that  SA  consists  of  three  levels.  Level  1 ,  perception,  involves  the 
accurate  identification  of  key  features  of  a  situation.  Understanding  what  these 
elements  mean,  or  comprehension,  defines  Level  2  SA.  Level  3  SA,  prediction, 
involves  the  ability  to  use  lower  levels  of  SA  to  make  accurate  projections  about  what  is 
likely  to  occur  in  the  near  future  in  a  given  situation.  Situation  awareness  is  viewed  as 
a  useful  construct  in  improving  decision-making  and  performance  in  a  variety  of 
domains  (Endsley,  Holder,  Leibrecht,  Garland,  Wampler,  &  Matthews,  2000). 

Until  recently,  however,  very  little  formal  attention  had  been  given  to  SA  for 
infantry  forces.  In  1998,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  hosted  a  workshop  designed  to  establish  dialogue  between  researchers 
and  infantry  warfighters,  identify  requirements  for  future  training,  leader  development, 
and  soldier  research  in  the  SA  domain,  and  to  establish  SA  requirements  and 
performance  measures  for  infantry  combatants  and  teams  (Graham,  1999).  A 
fundamental  recommendation  stemming  from  the  workshop  was  the  need  to  develop 
psychometrically  sound  measures  of  SA.  Reliable  and  valid  SA  measures  are  critical 
for  evaluating  the  impact  of  new  technology  on  performance  and  designing  information 
technology  systems  to  enhance  SA,  decision-making,  and  ultimately  soldier 
performance. 

Endsley  et  al.  (2000)  reviewed  SA  measurement  approaches,  and  evaluated 
them  for  applicability  to  the  infantry  environment.  According  to  Endsley  et  al.,  SA 
measurement  approaches  can  be  divided  into  four  types:  Process  indices,  direct 
measures,  behavioral  measures,  and  performance  measures.  Process  measures 
include  eye  movements,  communications,  and  verbalizations.  Direct  measures  include 
objective  measures  such  as  on-line  probes  and  “freeze”  probes,  and  subjective 
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measures  based  on  self  and  observer  ratings.  Behavioral  measures  involve  inferring 
SA  from  specific  behaviors  on  specific  subtasks,  such  as  “time  to  make  a  response 
(verbal  or  non-verbal)  to  some  event,  and  correct  or  incorrect  SA  as  identified  from 
soldier  verbalizations  and  appropriateness  of  a  given  behavior  for  a  particular  situation” 
(Endsley  et  al.,  2000,  p.  80).  Performance  measures  are  based  on  tactical 
performance  during  missions  and/or  exercises. 

Each  type  of  SA  measure  has  strengths  and  weaknesses  and  may  provide 
different  sorts  of  information.  For  example,  direct  subjective  SA  measures  have  an 
advantage  of  assessing  a  subject’s  personal  level  of  SA,  are  easily  administered,  and 
are  relatively  unobtrusive  to  collect.  However,  soldiers  may  not  know  what  information 
they  are  unaware  of  and  their  judgments  may  be  influenced  by  self-assessments  of 
their  own  or  their  unit’s  performance.  Similarly,  direct  objective  SA  measures,  such  as 
the  situation  awareness  global  assessment  technique  (SAGAT,  see  Endsley,  1995), 
have  the  advantages  of  providing  objective,  unbiased  estimates  of  SA,  but  are  relatively 
obtrusive  and  require  considerable  prior  analyses  to  develop  valid  measurement 
protocols.  However,  given  that  different  SA  measurement  approaches  may  provide 
different  types  of  information  or  be  more  acceptable  in  certain  situations,  a 
measurement  strategy  utilizing  multiple  SA  approaches  is  desirable. 

Based  on  Endsley  et  al.’s  (2000)  analysis,  Strater,  Endsley,  Pleban,  and 
Matthews  (2001)  developed  three  SA  measures  specifically  designed  to  measure  SA 
among  infantry  small  unit  leaders,  and  tested  them  during  platoon  level  missions  in  a 
virtual  environment.  In  this  study,  experienced  and  inexperienced  platoon  leaders  led 
three  squad  leaders  and  computer  generated  forces  through  four  missions  in  a  virtual 
environment  focusing  on  military  operations  in  urbanized  terrain  (MOUT).  The  first 
measure  was  a  SAGAT  protocol  modified  to  reflect  SA  requirements  for  infantry  platoon 
leaders  in  MOUT  missions.  The  second,  the  Situation  Awareness  Behaviorally 
Anchored  Rating  Scale  (SABARS),  was  a  direct  subjective  measure  utilizing  expert- 
observers  to  rate  the  platoon  leaders  on  behaviors  linked  to  SA  in  this  context.  The  last 
measure  developed  was  the  Participant  Situation  Awareness  Questionnaire  (PSAQ),  a 
subjective  SA  measure. 

The  SAGAT  procedure  involved  periodically  freezing  action  during  the  virtual 
MOUT  scenarios,  and  then  administering  probe  questions  to  the  platoon  leaders. 

These  questions  were  based  on  a  SA  requirements  analysis  completed  previously 
(Matthews,  Pleban,  Endsley,  &  Strater,  2000).  Based  on  this  requirements  analysis,  a 
list  of  21  queries  were  generated  that  indexed  information  critical  to  SA  for  small  unit 
leaders  in  missions  of  this  type.  These  queries  corresponded  to  the  three  levels  of  SA 
defined  by  Endsley  (1988).  In  the  virtual  environment,  unlike  the  real  world,  the 
“ground  truth”  (i.e.,  the  actual  status  of  these  factors,  which  can  be  verified  in  virtual 
simulations  and  carefully  conducted  field  experiments)  of  the  situation  is  known. 
Participants  whose  responses  match  the  ground-truth  criteria,  therefore,  presumably 
possess  higher  SA  than  those  whose  responses  fail  to  correspond  to  ground-truth,  thus 
providing  an  objective  index  of  the  individual’s  level  of  SA.  Results  showed  that  the 
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SAGAT  procedure  differentiated  SA  as  a  function  of  the  experience  level  of  the  platoon 
leaders.  It  also  was  sensitive  to  the  type  of  scenario  or  mission.  More  experienced 
platoon  leaders  showed  better  SA  for  enemy  information  such  as  location  and  strength, 
while  less  experienced  platoon  leaders  focused  their  attention  more  on  the  status  of 
their  own  forces. 

The  SABARS  also  showed  promise  for  measuring  SA.  Based  on  the  SA 
requirements  analysis  described  above  (Matthews  et  al.,  2000),  27  behaviors  and 
actions  linked  to  SA  in  MOUT  missions  were  identified.  Observer/controllers  (O/Cs) 
closely  watched  each  platoon  leader  work  through  the  four  missions.  The  O/Cs  then 
rated  the  platoon  leaders  on  a  five-point  scale  with  respect  to  how  well  they  performed 
these  actions.  Like  SAGAT,  the  SABARS  data  differentiated  SA  as  a  function  of  the 
experience  level  of  the  platoon  leaders.  Specifically,  experienced  platoon  leaders  were 
more  likely  to  gather  relevant  information,  follow  procedures,  and  to  focus  on  the  big 
picture  more  than  less  experienced  platoon  leaders. 

The  PSAQ,  which  consisted  of  three  questions  dealing  with  workload, 
performance,  and  awareness  of  the  situation  as  it  evolved,  showed  the  least  promise 
for  measuring  SA.  No  significant  effects  were  found  for  experience  level,  scenario,  or 
the  interaction  between  these  two  variables  (Strater  et  al.,  2001,  p.  26). 

The  importance  of  the  Strater  et  al.  (2001 )  experiment  is  that  it  represented  the 
first  systematic  attempt  to  assess  SA  for  typical  infantry  operations.  Both  SAGAT  and 
SABARS  showed  promise  for  measuring  SA  and  for  use  as  a  tool  in  estimating  the 
impact  of  new  technology  or  operational  procedures  on  small  leader  SA.  These 
measures  may  also  provide  a  basis  for  evaluating  strategies  designed  to  enhance  SA. 
Further  testing  of  both  of  these  measurement  approaches  in  other  mission  types  and  in 
field  training  exercises  are  necessary  to  fully  evaluate  their  psychometric  properties  and 
potential  utility  in  SA  research,  development,  and  training. 

Because  SAGAT  is  a  direct  objective  measure  of  SA,  it  might  seem  to  be  the 
preferred  method  for  measuring  SA.  In  simulations  and  other  tightly  controlled  settings 
it  may  indeed  represent  the  most  desirable  approach.  However,  much  of  the  training  in 
the  Army  is  done  in  the  field,  and  Army  leaders  and  trainers  often  are  reluctant  to 
interrupt  the  flow  of  an  exercise  to  allow  measures  of  what  they  may  view  as  elusive 
psychological  constructs.  In  these  settings,  the  SABARS  or  a  similar  approach  might 
be  more  desirable  to  the  extent  that  it  is  less  obtrusive.  However,  SABARS  is  labor 
intensive  in  the  sense  that  O/Cs  or  other  highly  experienced  personnel  must  be 
dedicated  to  observing  a  particular  leader  over  the  course  of  an  exercise  in  order  to 
provide  meaningful  ratings  of  SA  related  behaviors. 

Thus,  in  some  instances  a  subjective  SA  assessment  procedure  may  prove 
useful.  In  Strater  et  al.  (2001)  the  subjective  measure,  the  PSAQ,  did  not  provide 
useful  assessments  of  SA.  However,  numerous  other  direct,  subjective  SA  measures 
exist  and  have  been  used  in  a  variety  of  domains  (see  Endsley  et  al.,  2000).  More 
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recently,  McGuinness  and  Foy  (2000)  reported  the  development  of  a  subjective  SA 
measure  that  can  be  easily  tailored  to  a  variety  of  settings  or  domains.  McGuinnes  and 
Foy  refer  to  this  instrument  as  the  Crew  Awareness  Rating  Scale  (CARS).  The  CARS 
consists  of  two  sets  of  four  questions.  The  first  three  questions  of  each  set  correspond 
to  the  three  levels  of  SA  defined  by  Endsley  (1988),  i.e.,  perception,  comprehension, 
and  projection.  The  fourth  question  of  each  set  deals  with  how  well  the  respondent 
identifies  goals  for  the  situation  he  or  she  is  in.  Moreover,  the  first  set  of  four  questions 
pertain  to  assessing  SA  content,  for  example,  how  well  the  respondent  thinks  he  or  she 
understands  the  situation.  The  second  set  of  questions  addresses  workload,  for 
example,  how  much  mental  effort  is  required  to  achieve  understanding  in  a  given 
situation.  The  assessment  of  workload  is  an  important  aspect  of  SA.  There  could  be 
situations,  for  instance,  in  which  a  person  has  high  levels  of  SA,  but  most  of  their 
attentional  capacity  is  required  to  achieve  that  level  of  SA.  This  would  leave  little 
mental  workspace  left  to  allocate  to  other,  perhaps  equally  critical,  processes.  This  sort 
of  subjective  measure  could  prove  very  useful  in  evaluating  the  impact  of  new 
technologies  on  SA  and  mission  performance. 

The  purpose  of  the  current  study  was  to  adapt  CARS  for  application  to  small  unit 
leaders  in  the  Infantry.  The  revised  CARS  was  tested  during  a  virtual  MOUT 
experiment  initially  designed  to  evaluate  the  use  of  virtual  environments  for  night 
operations  training.  Specific  focus  was  directed  to  the  night  capabilities  currently 
available  on  selected  simulation  systems.  The  capabilities  evaluated  in  this  research 
included  night  vision  goggles  (Nags,  specifically  the  PVS-7B)  with  aiming  lights.  Two 
simulated  NVG  software  systems  were  evaluated;  one  system  was  developed  by 
Reality  by  Design  (RBD)  and  the  other  by  AcuSoft  (AS).  These  systems  were 
compared  to  actual  PVS-7Bs  that  were  worn  under  simulated  unaided  night  conditions. 
Previous  testing  showed  that  using  actual  PVS-7Bs  under  these  conditions  generated 
very  realistic  images. 

Because  the  focus  of  infantry  operations  is  on  team  conducted  missions,  the 
modified  CARS  is  referred  to  in  this  report  as  the  Mission  Awareness  Rating  Scale 
(MARS).  If  MARS  is  in  fact  sensitive  to  SA  content  and  SA  workload,  then  any 
differences  that  exist  among  technologies  for  simulating  night  environments  should  be 
reflected  in  MARS  ratings  to  the  extent  that  they  (1)  affect  the  user’s  ability  to  perceive 
elements  important  in  the  situation,  (2)  understand  the  meaning  of  those  elements,  and 
(3)  make  predictions  based  on  that  understanding  about  what  is  likely  to  occur  in  the 
near  future.  It  is  hypothesized  that  poorer  image  fidelity  should  either  decrement  SA 
content  and/or  increase  cognitive  workload  demands  needed  to  sustain  good  SA. 

Method 


Overview 


Four-man  infantry  teams  conducted  four  night  missions  in  an  immersive  virtual 
urban  environment.  Scenarios  were  generally  the  same,  differing  only  in  starting 
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location,  reconnaissance  objectives,  and  sequencing  of  the  appearance  of  civilians, 
enemy  forces,  vehicles,  aircraft,  etc.  Each  mission  was  conducted  under  one  of  four 
different  night  conditions.  Two  conditions  simulated  night  vision  goggles  (NVGs, 
specifically  PVS-7Bs)  and  aiming  lights  but  used  different  software  approaches  to 
create  their  effects.  Two  additional  conditions  simulated  an  unaided  night  environment, 
where  soldiers  wore  actual  NVGs  with  the  lens  cap  off  (to  approximate  real-world 
images  under  good  illumination),  or  with  lens  cap  on  (to  approximate  real-world  images 
under  bad  illumination).  [Note:  The  lens  cap  contains  a  tiny  pinhole  at  the  center  that 
greatly  restricts  the  amount  of  light  that  can  pass  through  the  system.].  A  retired  non¬ 
commissioned  officer,  who  role  played  the  squad  leader,  offered  minimal  guidance 
during  the  scenarios  and  provided  immediate  feedback  following  the  completion  of 
each  scenario  during  the  after  action  reviews.  ARI  researchers  served  as  additional 
observers  and  data  collectors. 

Participants 

Subjects  were  sixteen  enlisted  and  non-commissioned  officers  from  Fort 
Benning,  Georgia.  The  average  age  of  the  soldiers  was  25  years,  1 1  months.  Average 
time  in  service  was  75.2  months,  with  a  range  of  17  - 174  months.  Nine  soldiers  were 
Airborne  qualified  and  two  had  successfully  completed  Ranger  school.  Seven  soldiers 
had  completed  the  Primary  Leader  Development  Course  and  four  had  completed  the 
Basic  Non-Commissioned  Officer  Course. 

Soldiers  had  trained  at  the  McKenna  MOUT  site  at  Fort  Benning  an  average  of 
eight  times  since  basic  training  (range  0-50  times).  Only  one  soldier  had  ever 
operated  a  virtual  individual  combatant  simulator  such  as  the  type  employed  at  the 
Land  Warrior  Test  Bed  (LWTB).  The  majority  of  soldiers  had  some  experience  with 
military  simulation  systems.  Four  soldiers  had  trained  using  Simulation  Networking 
(SIMNET)  and/or  the  Close  Combat  Tactical  Trainer.  All  soldiers  had  previous 
experience  with  NVGs  and  aiming  lights. 

Instruments 


Biographical  Information  Questionnaire.  The  Biographical  Information 
Questionnaire  (Appendix  A)  is  a  multiple  choice/short  answer  paper-and-pencil 
instrument  designed  to  document  the  prior  military  training,  experience,  and  vision 
status  of  each  subject,  as  well  as  their  experience  with  computers  and  simulations. 

Virtual  Environment  Night  Operations  Comparison  Questionnaire.  At  the  end  of 
the  experiment  the  Virtual  Environment  Night  Operations  Comparison  Questionnaire 
was  administered  (Appendix  B).  Soldiers  rank  ordered  the  different  night  conditions 
based  on  how  well  each  environment  was  able  to  accurately  simulate  the  experience  of 
working  under  night  conditions.  Rankings  could  range  from  1  (Very  realistic  simulation 
of  night  conditions)  to  4  (Very  poor  simulation  of  night  conditions).  In  addition,  the 
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soldiers  responded  to  a  series  of  open-ended  questions  that  were  presented  orally  in 
individual  interviews  with  ARI  researchers.  Questions  addressed  the  following  issues: 

•  Rationale  for  ranking  a  particular  night  condition  as  the  best  (most  realistic)  or 

worst  (least  realistic) 

•  What  was  liked  the  most/least  about  these  night  simulations 

•  Whether  wearing  NVGs  increased  the  realism  of  the  simulation 

•  How  virtual  night  environments  could  be  incorporated  in  training 

Pre-Mission  Awareness  Rating  Scale  (Pre-MARS).  Pre-MARS  (Appendix  C)  is 
an  eight  question  instrument  based  on  the  Crew  Awareness  Rating  Scale  (CARS) 
described  in  depth  by  McGuinnes  and  Foy  (2000).  The  instrument  consists  of  four 
questions  designed  to  assess  the  ability  of  soldiers  to  perceive,  comprehend,  project, 
and  how  to  best  achieve  mission  goals  during  night  missions,  based  on  their 
experience  with  such  missions.  Four  additional  questions  address  the  same 
dimensions  (perceive,  comprehend,  project,  and  how  to  achieve  mission  goals)  but  ask 
the  respondent  how  much  mental  effort  (i.e.,  workload)  is  needed  to  perform  these 
functions  during  night  operations.  The  Pre-MARS  instrument,  therefore,  required  the 
participants  to  estimate,  based  on  their  past  experiences  with  night  missions,  their  level 
of  SA  and  amount  of  workload  required  to  achieve  that  level  of  SA. 

Mission  Awareness  Rating  Scale  (MARS).  The  MARS  instrument  (Appendix  D) 
included  the  same  eight  questions  as  the  Pre-MARS  instrument.  However,  instead  of 
basing  responses  on  general  past  experience  with  night  operations,  MARS  required  the 
participants  to  rate  their  SA  content  and  workload  for  the  virtual  scenario  they  had  just 
completed  in  the  LWTB.  Thus,  MARS  represents  a  subjective  self-assessment  of  SA 
and  SA-related  workload  for  each  of  the  four  specific  missions/night  environment 
conditions  included  in  the  current  experiment. 

Night  Scenarios 

All  scenarios  were  set  as  if  in  a  small  European  town.  The  town  was  a  virtual 
representation  of  the  McKenna  MOUT  training  site  at  Fort  Benning,  Georgia.  A  military 
subject  matter  expert  developed  four  versions  of  a  reconnaissance  mission.  Missions 
involved  eight  to  twelve  events.  Events  depicted  virtual  images  of  civilians, 
friendly/enemy  soldiers,  civilian  and  military  vehicles.  Entities  were  either  stationary  or 
moving.  These  events  were  scripted  to  occur  at  differing  times  as  the  soldiers  moved 
to  their  objective.  All  missions  concluded  with  a  firefight  between  the  test  subjects  and 
an  opposing  force  (OPFOR)  that  was  played  by  live  soldiers.  Missions  differed  in  the 
location  where  the  soldiers  started,  the  order  of  presentation  of  the  various  entities,  and 
building  location  of  the  OPFOR. 
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Apparatus 


Soldier  Visualization  Station  (SVS).  Four  full-immersion  SVS  systems  [helmet 
mounted  display  -  HMD  (this  feature  was  not  used),  weapon,  screen]  were  employed 
along  with  three  desktop  versions.  The  desktop  system  was  joystick  controlled.  The 
four  stand-alone  systems  were  linked  to  the  desktops.  Technical  specifications  of  the 
two  systems  are  shown  in  Table  1.  Team  members  could  communicate  with  each 
other  and  to  both  the  team  and  squad  leaders.  Only  the  team  leader  was  allowed  to 
communicate  with  the  squad  leader.  The  four  SVS  systems  were  housed  in  their  own 
enclosures.  These  enclosures  were  made  of  thick  black  cloth  and  fastened  to  a  metal 
frame  surrounding  the  SVSs.  They  were  designed  to  dampen  extraneous  sound, 
reduce  light,  and  minimize  distractions  from  other  people  moving  around  the  area. 

Table  1 

Technical  Specifications  of  the  SVS  and  Desktop  Systems 


System  Hardware 

(Immersive  and 
Desktop) 

•  Pentium  III  -  450  MHz  microprocessor 

•  128  Mb  RAM 

•  Obsidian  200  -  8440  3D  Graphics  Card 

•  SoundBlaster  AWE  64  Gold  Audio  Card 

•  Removable  4.55  GB  SCSI  Hard  Drive 

Movement  Control 

•  Weapon-mounted  thumbswitch 

•  Desktop  SVS  -  Microsoft  joystick  control 

Motion  Capture/ 
Weapon  Tracking 

•  InterSense  Mark2  X-Bar  Tracking  System 

•  Weapon  tracking  accurate  to  within  14  of  1° 

Visual  Display 

•  90°  x  60°  field  of  view  at  center  of  enclosure  (varies  with  position  change) 

•  Rear  screen  projection  resolution  1024  x  768 

•  Desktop  SVS  resolution  800  x  600 

Enclosures 

•  Aluminum  frame  over  black  sound-dampening  fabric.  (10  x  10  x  12) 

Software 

•  Reality  By  Design  proprietary  software 

Each  subject  operated  an  SVS,  while  the  squad  leader  and  the  OPFOR 
operated  the  desktop  systems.  ARI  researchers  observed  events  from  either  the 
simulation  system  operator’s  computer  screen  depicting  a  top-down  view  of  McKenna 
or  by  looking  at  the  squad  leader’s  screen.  The  squad  leader  and  the  simulation 
operator  systems  were  adjacent  to  each  other,  but  away  from  the  SVS  systems.  The 
OPFOR  systems  were  located  in  another  room  next  to  the  SVSs.  The  dimensions  and 
other  specific  details  of  its  layout  are  described  in  more  detail  by  Salter,  Eakin,  &  Knerr 
(1999). 

AN/PVS-7B  Night  Vision  Goggle.  The  A/N  (Army/Navy)  PVS-7B  NVG  is  a 
lightweight  image  intensification,  near  infrared  device  that  uses  ambient  light  conditions. 
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It  weighs  1.5  pounds  and  fastens  via  a  harness  to  the  user’s  head.  An  eyepiece 
diopter  is  provided  so  the  device  can  be  worn  without  corrective  lenses.  The  PVS-7B  is 
equipped  with  an  infrared  light  sources  and  positive  control  switch  that  permits  close-in 
viewing  (e.g.,  map  reading,  close-up  work  in  zero  ambient  light  conditions)  under 
limited  illumination.  (A  feature  not  used  in  the  current  experiment.)  An  auto  gain 
control  insures  the  right  level  of  illumination  regardless  of  light  sources  in  the  field  of 
view.  The  field  of  view  is  40  degrees  with  a  focus  range  of  six  inches  to  infinity.  Two 
AA  batteries  power  the  PVS-7B. 

Procedure 


Pilot  testing.  Four  individuals  with  NVG  experience  were  recruited  to  take  part  in 
the  pilot  testing  which  lasted  approximately  one  day.  These  individuals  were  briefed  on 
the  missions  and  conducted  one  movement  to  engagement  mission  per  night  condition 
as  a  fire  team.  Any  problems  with  a  particular  night  condition  or  the  NVGs  were  noted. 
All  procedures  including  real  time  data  collection  using  spot  report  check  lists  were 
examined.  The  squad  leader  rehearsed  his  role  with  the  other  team  members.  After 
the  testing  was  completed,  any  procedural  modifications  that  were  needed  were 
completed. 

Soldier  training.  Four  soldiers  arrived  each  morning  at  the  LWTB  and  were 
briefed  on  the  objectives  of  the  experiment.  They  were  given  a  chance  to  ask  any 
questions  concerning  their  roles  in  the  experiment.  They  then  completed  the 
Biographical  Information  Questionnaire  and  the  Pre-MARS. 

After  completing  the  questionnaires,  the  soldiers  were  given  a  brief  introduction 
to  the  SVS  system  and  allowed  hands-on  time  (approximately  30  minutes)  to  familiarize 
themselves  with  the  key  system  features  (e.g.,  moving  within  the  SVS  area,  moving  via 
the  thumb  switch  on  the  NM  rifle,  engaging  targets).  In  addition,  they  were  shown  what 
various  entities  looked  like  in  the  virtual  world  (e.g.,  buildings,  furniture,  friendly/enemy 
forces,  civilians,  vehicles,  and  aircraft)  under  simulated  NVG  conditions. 

Experimental  procedure.  Following  the  training  phase,  the  soldiers  composing 
the  fire  team  met  with  the  squad  leader  in  the  LWTB  conference  room.  The  squad 
leader  briefed  the  mission  to  the  team  members  who  were  given  a  chance  to  ask 
questions  and  then  allowed  15-20  minutes  to  develop  their  plan.  The  squad  leader 
emphasized  to  the  team  leader  that  spot  reports  (to  the  squad  leader)  were  required 
immediately  following  the  detection  of  any  event  (e.g.,  civilian  walking  across  the  street, 
burning  car,  etc.). 

The  soldiers  then  proceeded  to  the  simulator  bay  and  to  their  assigned 
immersible  SVS  systems.  The  squad  leader  moved  to  his  assigned  desktop  system  co¬ 
located  with  the  immersible  SVSs.  ARI  researchers  stayed  with  the  squad  leader. 

Each  researcher  had  headphones  and  was  able  to  hear  all  message  traffic  between  the 
team  and  squad  leaders.  In  addition,  they  could  also  observe  the  actions  of  the  team 
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from  the  screen  of  the  squad  leader’s  desktop  SVS.  The  OPFOR  were  already  in  place 
in  front  of  their  desktop  systems  in  a  separate  room.  After  completing  system  checks 
on  the,SVSs  and  the  communication  nets,  the  scenario  started. 

Presentation  of  the  four  night  conditions  was  balanced  using  a  Latin  square 
design.  After  each  night  condition/mission,  the  squad  leader  conducted  a  brief  After 
Action  Review.  The  soldiers  then  completed  the  MARS  instrument. 

This  sequence  was  presented  a  total  of  four  times.  At  the  conclusion  of  the 
experiment,  the  soldiers  completed  the  first  part  of  the  Virtual  Environment  Night 
Operations  Comparison  Questionnaire  (rank  ordering  of  night  conditions).  Individual 
structured  interviews  were  then  conducted  based  on  the  remaining  items  making  up  the 
questionnaire. 

Four  night  environment  conditions  were  examined.  The  first  condition  (“RBD”) 
was  a  night  scene  software  projection  developed  by  Reality  by  Design  (RBD),  designed 
to  simulate  the  images  soldiers  would  see  while  using  night  vision  goggles.  In  this 
condition,  no  image  intensification  device  was  worn  by  the  participants.  The  second 
condition  (“AS”)  was  a  night  simulation  software  projection  designed  by  AcuSoft  (AS). 
Like  the  RBD  condition,  this  software  simulated  night  images  without  the  use  of 
goggles.  The  third  condition  (“Caps  Off’)  required  participants  to  wear  PVS-7B  NVGs 
with  the  lens  cap  off  while  viewing  the  unaided  night  projections  (that  is,  what  a  soldier 
would  see  at  night  without  any  image  intensification  device).  This  is  the  manner  that 
NVGs  are  usually  used  in  the  field.  The  final  condition  (“Caps-On”)  involved  wearing 
the  PVS-7Bs  with  the  lens  cap  on  while  viewing  the  same  unaided  projection.  The  caps 
allow  a  minimal  amount  of  light  in  via  a  pinhole.  This  greatly  reduced  the  visibility  of  the 
scene. 


Results 

Pre-MARS.  Pre-MARS  was  designed  to  provide  an  estimate  of  SA  during  night 
operations  based  on  the  participant’s  general  experience  in  night  operations.  Results 
from  the  Pre-MARS  instrument  are  summarized  in  Table  2,  which  shows  the  mean  and 
standard  deviation  for  the  Pre-MARS  SA  content  and  SA  workload  subscales.  A  one¬ 
way  repeated  measures  analysis  of  variance  (ANOVA)  revealed  a  significant  difference 
among  the  four  subscales  for  SA  content  (F[3,45]  =  3.17,  p  <  .05,  eta  squared  =  .17), 
with  the  most  demanding  SA  component  being  level  3  SA,  prediction.  To  determine 
which  means  differed  significantly,  post  hoc  analyses  were  conducted  using  paired  t 
tests,  with  alpha  set  at  .05,  df  =  15,  and  using  the  Bonferroni  correction  to  account  for 
multiple  comparisons.  This  showed  that  only  the  predict  versus  decide  comparison  was 
significant.  Similarly,  responses  to  the  SA  workload  subscales  also  differed 
significantly,  03,45]  =  6.37,  p  <  .05,  eta  squared  =  .30.  Post  hoc  comparisons,  again 
using  paired  t  tests,  showed  that  perceive  versus  predict  and  comprehend  versus 
predict  were  significant. 
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Table  2 


Pre-MARS  Ratings  of  Night  Mission  SA 


Mean 

Standard 

Deviation 

N 

SA  Content* 

Perceive 

1.94 

.68 

16 

Comprehend 

1.63 

.62 

Predict 

2.13 

.72 

Decide 

1.62 

.72 

SA  Workload" 

Perceive 

1.81 

.66 

16 

Comprehend 

1.87 

.62 

Predict 

2.44 

.81 

Decide 

2.13 

.72 

*  F  (3,45)  =  3.17,  £  <  .05,  eta  squared  =  .17 
"F  (3,45)  =  6.37,  p  <  .05,  eta  squared  =  .30 

MARS.  Table  3  shows  the  means  and  standard  deviations  for  MARS  SA  content 
subscales  for  each  of  the  treatment  conditions,  Caps  Off,  Caps  On,  RBD,  and  AS.  For 
SA  content,  the  participants  rated  the  Caps  On  condition  as  being  the  most  difficult  for 
all  four  subscales.  Moreover,  comparing  within  the  Caps  On  condition  across  the 
subscales,  the  most  difficult  tasks  were  perception  followed  closely  by  prediction.  For 
the  SA  content  subscales,  the  least  demanding  ways  of  simulating  NVG-like  images  in 
a  virtual  environment  were  RBD  and  AS  for  all  four  subscales.  The  Caps  Off  condition 
fell  intermediate  in  difficulty  for  all  four  subscales,  with  mean  ratings  similar  to  those 
from  the  Pre-MARS  scale. 

Four  one  way  repeated  measures  ANOVAs  were  conducted  for  each  SA  content 
subscale.  The  ANOVAs  were  statistically  significant  with  large  eta  squares.  Post-hoc 
comparisons  within  each  level  of  SA  content  were  performed  using  paired  t  tests  with 
alpha  set  at  .05  and  14  degrees  of  freedom.  Using  the  Bonferroni  correction  to  adjust 
for  multiple  comparisons,  all  pair-wise  comparisons  for  level  1  SA  (perceive),  with  the 
exception  of  RBD  versus  AS,  were  significant.  For  level  2  SA  ( comprehend ),  all  pair¬ 
wise  comparisons  were  significant  except  for  Caps  Off  versus  RBD,  and  RBD  versus 
AS.  Among  level  3  SA  ( prediction )  comparisons,  the  following  were  statistically 
significant:  Caps  On  versus  Caps-Off,  Caps  On  versus  AS,  and  Caps  On  versus  RBD. 

No  other  pair-wise  comparisons  were  significant  for  level  3  SA.  Finally,  for  the  decide 
questions  under  SA  content,  three  comparisons  were  statistically  significant:  Caps  Off 
versus  AS,  Caps  On  versus  AS,  and  Caps  On  versus  RBD. 
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Table  3 


Means  and  Standard  Deviations  for  MARS  SA  Content  Subscales  for  Each  Night 


Simulation  Condition 


SA  Content 

Perceive1 

Comprehend2 

Predict3 

Decide4 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

Caps  On 

2.73 

.88 

15 

2.27 

.70 

15 

2.47 

.83 

15 

2.13 

.64 

15 

Caps  Off 

1.80 

.77 

15 

1.60 

.63 

15 

1.87 

.64 

15 

1.80 

.68 

15 

RBD 

1.13 

.35 

15 

1.20 

.41 

15 

1.53 

.52 

15 

1.27 

.46 

15 

AS 

1.13 

.35 

15 

1.20 

.41 

15 

1.60 

.74 

15 

1.20 

.41 

15 

1 

2 

3 

4 


F  (3,  42)  =  27.36,  p<. 01, 
F  (3,  42)=  15.06,  p<  .01, 
F  (3,  42)  =  1 0.48,  p  <  .01 , 
F  (3,  42)  =  10.36,  £<.01, 


eta  squared  =  .66 
eta  squared  =  .52 
eta  squared  =  .43 
eta  squared  =  .42 


For  the  SA  workload  scales  a  similar  pattern  emerged.  Repeated  measures 
ANOVAs  revealed  significant  differences  in  rated  difficulty  among  the  four  ways  of 
simulating  NVG  image  effects  across  all  four  subscales,  with  the  Caps  On  condition 
being  rated  the  most  difficult  for  each  subscale.  With  that  condition,  the  most  difficult 
tasks  were  prediction,  followed  closely  by  perception.  Again,  the  easiest  conditions  for 
all  four  subscales  were  RBD  and  AS.  The  Caps  Off  ratings  again  fell  between  the  Caps 
On  and  RBD/AS  ratings,  and  were  similar  to  those  obtained  from  Pre-MARS  ratings  of 
SA  workload  for  night  operations  in  the  real  world.  Eta  squared  values  were  also  robust 
for  these  comparisons,  indicating  a  strong  relationship  between  workload  subscales 
and  treatment  condition.  These  data  are  summarized  in  Table  4. 


Table  4 

Means  and  Standard  Deviations  for  MARS  SA  Workload  Subscales  for  Each  Night 
Simulation  Condition 


SA  Workload 

Perceive1 

Comprehend2 

Predict3 

Decide4 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

Caps  On 

2.73 

.80 

15 

2.47 

.52 

15 

2.80 

.68 

15 

2.27 

.80 

15 

Caps  Off 

2.00 

1.00 

15 

1.67 

.82 

15 

2.00 

.76 

15 

2.07 

.70 

15 

RBD 

1.40 

.51 

15 

1.13 

.35 

15 

1.53 

.64 

15 

1.33 

.62 

15 

AS 

1.40 

.51 

15 

1.40 

.63 

15 

1.53 

.74 

15 

1.47 

.64 

15 

.63 

1 F  (3,  42)  =  12.1 

33,  p< 

.01 ,  eta  squared  = 

.48 

2  F  (3,  42)  =  24.40,  p  <  .01 ,  eta  squared  =  .64 

3  F  (3,  42)  =  14.58,  p  <  .01,  eta  squared  =.51 

4  F  (3,  42)  =  9.59,  p  <  .01,  eta  squared  =  .41 
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Pair-wise  t  tests  were  again  used  for  the  four  SA  Workload  scales  as  a  post-hoc 
procedure  to  locate  which  means  within  comparisons  were  statistically  significant.  With 
alpha  set  at  .05,  14  degrees  of  freedom,  and  using  the  Bonferroni  correction,  only  two 
comparisons  were  found  to  be  significant  for  the  perceive  scale:  Caps-On  versus  AS, 
and  Caps-On  versus  RBD.  For  the  comprehend  and  predict  scales  there  were  three 
significant  comparisons:  Caps  off  versus  Caps  on,  Caps  On  versus  AS,  and  Caps  On 
versus  RBD.  Finally,  for  the  decide  scale,  significant  differences  were  found  between 
Caps  versus  AS,  Caps  Off  versus  RBD,  Caps  On  versus  AS,  and  Caps  On  versus 
RBD. 


MARS  subscale  intercorrelations.  Conceptually,  there  should  be  a  high  positive 
correlation  among  the  MARS  subscales.  For  example,  good  level  2  SA  should  be 
dependent  upon  achieving  high  level  1  SA.  Appendix  E  shows  the  MARS  subscale 
intercorrelations  for  the  Caps  Off,  Caps  On,  RBD,  and  AS  conditions,  respectively. 
These  tables  reveal  large  and  statistically  significant  (£  <  .05)  correlations  between 
subscales  for  all  but  the  RBD  condition.  For  that  condition,  only  6  of  the  28  possible 
correlations  were  statistically  significant. 

Subjective  data.  The  soldiers  participating  in  the  experiment  were  asked  to  rank 
order  (from  best  to  worst)  how  well  each  method  was  able  to  accurately  simulate  the 
experience  of  working  under  night  conditions.  Forty  percent  of  the  soldiers  rated  the 
Caps  Off  condition  as  most  similar  to  actual  night  conditions,  followed  by  Caps  On  and 
RBD  (27%  each).  Only  one  soldier  rated  AS  as  the  most  realistic  condition.  Two 
conditions  were  identified  as  being  the  poorest  approximation  of  night  conditions,  with 
53%  of  the  soldiers  rating  the  AS  condition  as  the  worst  approach,  followed  closely  by 
47%  who  rated  the  Caps  On  as  the  worst. 

Discussion 

The  main  experimental  question  addressed  was  whether  MARS  would 
discriminate  among  four  approaches  to  simulating  night  vision  goggle  image  effects. 
The  results  indicate  that  it  did  detect  differences  both  in  the  SA  content  and  SA 
workload  as  a  function  of  the  simulation  procedure.  The  MARS  instrument  also  may 
provide  insight  on  which  method  most  closely  simulates  NVG  images  seen  under  real 
world  conditions  using  PVS-7Bs.  For  example,  the  Caps  Off  condition  emerged  with 
ratings  on  all  subscales  that  were  closest  in  value  to  those  obtained  from  the  Pre- 
MARS  instrument.  One  of  the  conditions  -  Caps  On  -  was  consistently  rated  as  more 
demanding  than  actual  night  operations.  Two  others,  the  RBD  and  the  AS  conditions, 
were  uniformly  rated  as  easier  than  actual  night  operations.  The  question  of  which 
night  simulation  is  most  “realistic”  is  complicated,  with  many  image  and  fidelity  issues 
involved,  but  SA  as  indexed  by  MARS  may  represent  one  component  used  to  judge  the 
realism  of  these  night  environments. 

The  second  line  of  evidence  supporting  this  conclusion  was  in  the  comparison  of 
MARS  responses  with  feedback  given  to  the  experimenters  by  the  soldiers  at  the 
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conclusion  of  the  experiment.  Forty  percent  of  the  soldiers  rated  the  Caps  Off  condition 
as  the  most  realistic  NVG  environment.  In  contrast,  none  reported  this  condition  as 
being  the  worst. 

Thus,  it  appears  that  MARS  discriminated  among  these  four  treatment 
conditions.  It  could  be  questioned,  however,  what  exactly  it  is  that  MARS  measures.  In 
order  to  demonstrate  that  MARS  actually  measures  SA,  per  se,  it  will  be  necessary  to 
compare  MARS  responses  against  a  more  objective  and  better  validated  SA  measure. 
Probably  the  best  of  these  measures  is  SAGAT.  In  contrast  to  MARS,  which  relies  on 
subjective  and  general  ratings  of  SA  content  and  SA  workload,  SAGAT  involves  specific 
questions  on  SA  related  information,  such  as  the  location  of  friendly  and  enemy  troops, 
status  of  ammunition,  fuel,  and  supplies,  etc.  These  responses  are  compared  to 
“ground  truth.”  To  establish  that  MARS  is  measuring  SA,  and  not  some  other  process, 
it  will  be  necessary  to  simultaneously  collect  MARS  and  SAGAT  data,  using  the  latter 
as  the  validation  criterion. 

With  the  exception  of  the  RBD  condition,  the  MARS  subscales  generally  showed 
large  and  positive  inter-correlations.  According  to  Endsley  et  al.  (2000)  model  of 
infantry  SA,  this  would  be  expected  to  be  the  case.  It  would  be  difficult,  according  to 
this  model,  to  have  high  level  3  SA  (projection)  if  lower  levels  of  SA  were  not  also  at  a 
high  level.  The  failure  to  find  high  inter-correlations  among  the  MARS  subscales  for  the 
RBD  condition  may  reflect  some  sort  of  “floor”  effect.  Both  SA  content  and  workload 
were  rated  as  relatively  easy  to  obtain  under  that  condition,  and  this  could  have 
resulted  in  a  case  where  good  higher-order  SA  might  not  depend  on  good  lower  level 
SA.  The  AS  condition  had  similar  SA  content  and  workload  ratings  as  the  RBD 
condition,  however,  yet  showed  relatively  large  subscale  inter-correlations.  So  it  is 
possible  that  some  other  unique  aspect  of  the  RBD  condition  affected  the  ratings  for 
that  condition. 

The  importance  of  developing  a  psychometrically  sound  subjective  SA 
measurement  approach  that  could  be  used  in  a  variety  of  Army  field  settings  is  that 
more  objective  approaches,  like  SAGAT,  are  viewed  as  too  obtrusive  to  be  used  in  field 
exercises.  Infantry  teams  operate  in  a  fluid  and  dynamic  way,  and  Army  trainers  are 
concerned  that  stopping  action  -  even  if  for  only  a  few  minutes  -  will  detract  from  the 
realism  of  the  training.  However,  some  method  for  assessing  SA  during  these  types  of 
exercises  is  necessary  for  a  number  of  reasons.  First,  new  technologies  designed  to 
improve  communication  and  enhance  information  flow  may  actually  detract  from  SA  by 
causing  overload.  Second,  there  needs  to  be  a  way  of  assessing  new  training 
strategies  on  SA.  Finally,  because  good  decision-making  hinges  on  good  SA, 
techniques  and  approaches  designed  to  improve  SA  itself  must  be  evaluated.  While 
some  of  these  objectives  can  be  accomplished  in  virtual  environments  or  very  carefully 
controlled  and  conducted  field  training,  much  of  this  will  be  conducted  in  settings  where 
SAGAT  or  similar  procedures  would  be  found  unacceptable. 
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In  conclusion,  MARS  represents  a  potentially  useful  instrument  for  use  in 
assessing  subjective  SA.  The  instrument  could  be  used  to  assess  the  impact  of  new 
information  technologies  on  SA  content  and  SA  workload,  and  thus  help  determine  if 
these  technologies  actually  facilitate  or  inhibit  decision-making  and  performance. 

Future  research  efforts  should  center  on  validating  MARS  against  more  objective  direct 
measures  of  SA,  such  as  SAGAT,  and  evaluating  its  usability  and  psychometric 
properties  in  the  field  environment. 
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Appendix  A 

Biographical  Information  Questionnaire 


Biographical  Information  Questionnaire 


Name  _  Unit _  Date _ 

Please  fill  in  the  blank  or  mark  or  circle  the  appropriate  response. 

1 .  What  is  your  age? _ Years  _ Months 

2.  MOS _ 

3.  Rank _ 

4.  Time  in  service  Years _  Months _ 

5.  What  is  your  current  (or  most  recent)  duty  position?  _ 

How  long  in  this  position?  _ 

6.  What  Army  training  courses  have  you  completed?  Check  all  that  apply. 

_ PLDC  _ BNCOC  _ ANCOC  _ _ _  Airborne 

_ BFV  Leader  Course  _ Ranger  _ Air  Assault 

_ Other  (please  specify) _ _ _ 

7.  How  susceptible  to  motion  or  car  sickness  do  you  feel  you  are? 


1  2 

3  4  5 

6 

7 

not 

moderately 

highly 

susceptible 

susceptible 

susceptible 

8.  Do  you  have  normal  20/20  vision  without  glasses?  _ Yes _ No 

9.  Do  you  have  20/20  vision  with  contact  lenses  or  glasses? _ Yes  _ 

_ NA 

10.  Are  you  color  blind?  _ Yes  _ No 

1 1 .  Are  you  _ right  handed? _ left  handed? 


No 
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r 


12.  How  many  hours  per  week  do  you  play  ‘virtual  reality’  type  games? _ hours 

per  week 

13.  How  often  have  you  trained  at  the  McKenna  MOUT  site  (not  including  demos)? 
_ times 

14.  Have  you  ever  been  in  a  Virtual  Individual  Combatant  (VIC)  simulator  at  the  Land 
Warrior  Test  Bed  before? 

_ Yes  _ No 

If  YES,  which  one(s)?  (Describe  if  you  cannot  remember  the  name) 

15.  Have  you  had  any  other  experience  with  military  computer  simulations? 

_ Yes  _ No 

If  yes,  please  describe  briefly  or  give  the  names  of  the  simulators. 
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Appendix  B 

Virtual  Environment  Night  Operations  Comparison  Questionnaire 
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Virtual  Environment  Night  Operations  Comparison  Questionnaire 

Name: _  Date: _  VE  Comparison 

1 .  Rank  order  the  virtual  night  environments  from  1-4,  based  on  how  well  each  was 
able  to  accurately  simulate  the  experience  of  working  under  night  conditions. 
(Note.  Use  each  number  - 1,  2,  3, 4  only  once  in  your  rankings). 

1  =  Best  (Very  realistic  simulation  of  night  conditions) 

4  =  Worst  (Very  poor  simulation  of  night  conditions) 

_ PVS-7B  Good  Illumination 

_ PVS-7B  Bad  Illumination 

_ RBD  NVG 

AS  NVG 


Structured  Interview  Questions 

2.  Why  did  you  select  the _ system  as  the  best? 

3.  Why  did  you  select  the _ system  as  the  worst? 

4.  What  did  you  like  most  about  these  night  simulations?  Why? 

5.  What  did  you  like  least  about  these  night  simulations?  Why? 

6.  Did  wearing  NVGs  (PVS-7Bs)  increase  the  realism  of  the  simulation?  If  so,  how? 

7.  How  would  you  incorporate  virtual  night  environments  in  your  training? 
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Appendix  C 

Pre-Mission  Awareness  Rating  Scale  (Pre-MARS) 


Name _  Date _ 

Position  (check  one) _ Team  Leader  _ Team  Member 


Night  Operations  Situation  Awareness  Questionnaire 

The  following  four  questions  deal  with  your  ability  to  perceive  and  understand 
important  aspects  of  a  situation  during  night  operations  in  the  REAL  WORLD 
while  using  NVG  (&  aiming  light)  or  other  equipment  you  routinely  use: 


1 .  Please  rate  your  ability  to  identify  mission-critical  cues  at  night. 

_  very  easy  -  able  to  identify  all  cues 

_ fairly  easy  -  can  identify  most  cues 

_  somewhat  difficult  -  many  cues  hard  to  identify 

_  very  difficult  -  have  substantial  problems  identifying  most  cues 

2.  How  well  can  you  understand  what  is  happening  at  night? 

_ very  well  -  can  always  understand  the  situation 

fairly  well  -  can  understood  most  aspects  of  the  situation 

_  somewhat  poorly  -  often  have  difficulty  understanding  the  situation 

_ very  poorly  -  the  situation  usually  does  not  make  sense  to  me 

3.  How  well  can  you  predict  what  is  about  to  occur  next  at  night? 

_ very  well  -  can  predict  with  accuracy  what  is  about  to  occur 

_ fairly  well  -  can  make  accurate  predictions  most  of  the  time 

_  somewhat  poor  -  misunderstand  the  situation  much  of  the  time 

_ very  poor  -  usually  unable  to  predict  what  is  about  to  occur 

4.  How  aware  are  you  of  how  to  best  achieve  your  goals  at  night? 

_ very  aware  —  know  how  to  achieve  goals  at  all  times 

_ fairly  aware  -  know  most  of  the  time  how  to  achieve  mission  goals 

_  somewhat  unaware  -  am  not  aware  of  how  to  achieve  some  goals 

_ very  unaware  -  generally  unaware  of  how  to  achieve  goals 
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The  last  four  questions  ask  how  difficult  it  is  for  you  to  detect  and  understand 
important  cues  present  at  night  in  the  REAL  WORLD. 

5.  How  difficult  -  in  terms  of  mental  effort  required  -  is  it  for  you  to  identify  or  detect 
mission-critical  cues  at  night? 

_  very  easy  -  can  identify  relevant  cues  with  little  effort 

_ fairly  easy  -  can  identify  relevant  cues,  but  some  effort  required 

_  somewhat  difficult  -  some  effort  is  required  to  identify  most  cues 

_ very  difficult  -  substantial  effort  is  required  to  identify  relevant  cues 

6.  How  difficult  -  in  terms  of  mental  effort  -  is  it  to  understand  what  is  going  on  at 
night? 


_ very  easy  -  understand  what  goes  on  with  little  effort 

_ fairly  easy  -  understand  events  with  only  moderate  effort 

_  somewhat  difficult  -  hard  to  comprehend  some  aspects  of  situation 

_ very  difficult  -  hard  to  understand  most  or  all  aspects  of  situation 

7.  How  difficult  -  in  terms  of  mental  effort  -  is  it  to  predict  what  is  about  to  happen  at 
night? 


_  very  easy  -  little  or  no  effort  needed 

_ fairly  easy  -  moderate  effort  required 

_  somewhat  difficult  -  many  projections  require  substantial  effort 

_ very  difficult  -  substantial  effort  required  on  most  or  all  projections 

8.  How  difficult  -  in  terms  of  mental  effort  -  is  it  to  decide  on  how  to  best  achieve 
mission  goals  at  night? 

_ very  easy  -  little  or  no  effort  needed 

_ fairly  easy  -  moderate  effort  required 

_  somewhat  difficult  -  substantial  effort  needed  on  some  decisions 

_ very  difficult  -  most  or  all  decisions  require  substantial  effort 
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Appendix  D 

Mission  Awareness  Rating  Scale  (MARS) 
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Name _  Date _ 

Position  (check  one)  _ Team  Leader  _ Team  Member 

Treatment  Condition:  _  PVS-7/GI  _ PVS-7/BI  _ RBS/NVG  _  ASW/NVG 


Mission  Awareness  Rating  Scale  (MARS) 

Instructions.  Please  answer  the  following  questions  about  the  exercise  you  just 
completed.  Your  answers  to  these  questions  are  important  in  helping  us  evaluate  the 
effectiveness  of  this  training  exercise.  Check  the  response  that  best  applies  to  your 
experience. 

The  first  four  questions  deal  with  your  ability  to  detect  and  understand  important 
cues  present  during  the  exercise. 

1 .  Please  rate  your  ability  to  identify  mission-critical  cues  in  this  exercise. 

_ very  easy  -  able  to  identify  all  cues 

_ fairly  easy  -  could  identify  most  cues 

_ somewhat  difficult  -  many  cues  hard  to  identify 

_ very  difficult  -  had  substantial  problems  identifying  most  cues 

2.  How  well  did  you  understand  what  was  going  on  during  the  exercise? 

_ very  well  -  fully  understood  the  situation  as  it  unfolded 

_ fairly  well  -  understood  most  aspects  of  the  situation 

_ somewhat  poorly  -  had  difficulty  understanding  much  of  the  situation 

_ very  poorly  -  the  situation  did  not  make  sense  to  me 

3.  How  well  could  you  predict  what  was  about  to  occur  next  in  the  exercise? 

_ very  well  -  could  predict  with  accuracy  what  was  about  to  occur 

_ fairly  well  -  could  make  accurate  predictions  most  of  the  time 

_  somewhat  poor  -  misunderstood  the  situation  much  of  the  time 

_ very  poor  -  unable  to  predict  what  was  about  to  occur 

4.  How  aware  were  you  of  how  to  best  achieve  your  goals  during  this  exercise? 

_  very  aware  -  knew  how  to  achieve  goals  at  all  times 

_ fairly  aware  -  knew  most  of  the  time  how  to  achieve  mission  goals 

_ somewhat  unaware  -  was  not  aware  of  how  to  achieve  some  goals 

_ very  unaware  -  generally  unaware  of  how  to  achieve  goals 
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The  last  four  questions  ask  how  difficult  it  was  for  you  to  detect  and  understand 
important  cues  present  during  the  exercise. 


5.  How  difficult  -  in  terms  of  mental  effort  required  -  was  it  for  you  to  identify  or  detect 
mission-critical  cues  in  the  exercise? 

_  very  easy  -  could  identify  relevant  cues  with  little  effort 

_ fairly  easy  -  could  identify  relevant  cues,  but  some  effort  required 

_  somewhat  difficult  -  some  effort  was  required  to  identify  most  cues 

_  very  difficult  -  substantial  effort  required  to  identify  relevant  cues 

6.  How  difficult  -  in  terms  of  mental  effort  -  was  it  to  understand  what  was  going  on 
during  the  exercise? 

_  very  easy  -  understood  what  was  going  on  with  little  effort 

_ fairly  easy  -  understood  events  with  only  moderate  effort 

_  somewhat  difficult  -  hard  to  comprehend  some  aspects  of  situation 

_ very  difficult  -  hard  to  understand  most  or  all  aspects  of  situation 

7.  How  difficult  -  in  terms  of  mental  effort  -  was  it  to  predict  what  was  about  to  happen 
during  the  exercise? 

_  very  easy  -  little  or  no  effort  needed 

_ fairly  easy  -  moderate  effort  required 

_  somewhat  difficult  -  many  projections  required  substantial  effort 

_  very  difficult  -  substantial  effort  required  on  most  or  all  projections 

8.  How  difficult  -  in  terms  of  mental  effort  -  was  it  to  decide  on  how  to  best  achieve 
mission  goals  during  this  exercise? 

_ very  easy  -  little  or  no  effort  needed 

_ fairly  easy  -  moderate  effort  required 

_  somewhat  difficult  -  substantial  effort  needed  on  some  decisions 

_ very  difficult  -  most  or  all  decisions  required  substantial  effort 
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Appendix  E 

MARS  Subscale  Correlation  Matrices 
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Caps  Off  Condition 
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Perceive 

Content/ 

Comprehe 

nd 

Content/ 

Project 

Content/ 

Decide 

Workload/ 
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SO 

* 

* 

Os 
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* 

* 

so 

o 

r- 
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* 
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* 
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r- 

Tt 
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* 

* 

in 

c- 

00 

* 

* 
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00 
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* 
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Os 
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* 

* 

co 

so 

SO 
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Caps  On  Condition 
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nd 
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Project 
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Perceive 
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Workload/ 
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ON 
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* 

* 

00 

o 

Os 

* 

* 

<N 
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* 

* 

Os 

<N 
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* 

* 

VO 
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Decide 

.492 

.657** 
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* 

* 

VO 
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* 

* 
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RBD  Condition 
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VO 
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AS  Condition 
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* 
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* 
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* 

* 
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* 
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* 
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* 

* 
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Project 
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